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ABSTRACT 


The  purpose  of  this  report  is  to  permit  management  to 
quickly  and  inexpensively  evaluate  a  real  time  data  process- 
ing  system  and  to  express  a  statistical  confidence  in  the 
validity  of  their  evaluation. 


INTRODUCTION 

This  report  results  from  a  conclusion,  by  the  authors,  that  there 
presently  exist  no  standards  for  the  manner  and  procedures  by  which  a  real 
time  electronic  computer  data  processing  system  can  be  evaluated. 

The  modem  world  is  in  a  dynamic  era  of  technological  advance.  Each 
program  system  developed  for  the  processing  of  data  on  electronic  computers 
appears  to  be  soon  replaced  by  a  larger  or  '*better”  system  that  shows  promise 
of  doing  the  required  job  faster  and  better.  It  is  no  wonder,  then,  that 
an  ever  increasing  burden  is  placed  upon  management  in  the  quest  to  seek 
the  "best"  solution  to  the  computer  problems  that  face  it.  Unfortunately, 
however,  this  attenpt  to  select  the  best  solution  often  involves  the  choice 
of  computer  programs  that  are  far  too  complex  and  intricate  to  be  grasped 
in  their  entirety.  Different  language  is  involved,  the  sciences  are  called 
upon  constantly  and  even  the  computer  people  become  a  "breed  unto  them¬ 
selves."  More  often  than  not,  then,  management  is  called  upon  to  accept  as 
gospel  a  processing  system  that  they  cannot  hope  to  coapletely  understand, 
much  less  to  evaluate.  The  added  factor  that  the  development  of  data  pro¬ 
cessing  often  requires  many  months  of  painstaking  labor  increases  manage¬ 
ment's  burden  by  requiring  them  to  determine  the  worth  of  a  system  long 
after  the  original  specifications  have  been  written.  The  net  result  is  the 
rather  ludicrous  situation  which  requires  management  to  ^prove  (or  in  some 
cases  buy)  a  system  which  is  too  complicated  to  understand  completely  with¬ 
out  extensive  e]q>osure,  yet  which  may  contain  shortcomings  when  evaluated 
in  the  light  of  current  operational  requirements. 

Such  a  situation  may  or  may  not  be  critical,  depending  on  the  type  of 
data  that  is  being  processed  and  the  requirements  of  management.  Certainly, 
however,  it  can  be  seen  that  in  an  area  such  as  the  real  time  processing 
of  strategical  and  tactical  information  iq>on  which  military  commanders  will 
make  their  decision,  an  incorrect  choice  of  a  system  can  prove  disastrous 


(or  at  the  very  least,  embarrassing).  It  is  also  evident  that  regardless 
of  whether  the  system  involves  the  military  or  not,  the  possibility  of  an 
incorrect  choice  is  certainly  undesirable. 

The  answer  to  the  problem,  as  we  find  it  then,  is  to  provide  management 
with  the  tools  by  «diich  they  can  effectively  evaluate  a  proposed  (or  oper¬ 
ating)  real  time  data  processing  system  in  a  practical  manner  without  the 
necessity  of  eii|>loying  a  large  groiq>  of  specialists  specifically  for  this 
purpose . 

Recently  a  problem  of  the  nature  discussed  was  assigned  to  the  authors; 
i.e.,  to  evaluate  the  effectiveness  of  a  large  digital  conputer  program  (to 
see  if  it  did  in  fact  do  the  job  which  it  was  sig>posed  to  do).  It  became 
apparent  as  the  preliminary  planning  of  the  evaluation  progressed  that  there 
was  very  little  written  in  this  area  that  would  provide  guidelines  by  which 
the  evaluation  could  be  acconplished.  It  is  true  that  much  has  been  written 
about  the  theory  of  evaluation  and  that  similarly  many  large  scale  evaluations 
have  been  acconplished  at  great  expense  and  with  considerable  time  involved. 
While  such  theory  and  such  evaluations  have  been  of  value  in  the  past  and 
will  certainly  continue  to  be  of  value  in  the  future,  it  would  seem  that 
management  has  a  right  to  expect  that  they  can  judge  a  real  time  system 
quickly  and  easily  with  the  use  of  relatively  inexperienced  personnel  now 
that  coDf>uter  processing  has  become  an  integral  part  of  their  operations. 

This,  then,  is  the  purpose  of  this  report,  that  is,  to  permit  management  to 
quickly  and  inexpensively  evaluate  a  real  time  system  and  to  express  a  con¬ 
fidence  in  the  validity  of  their  evaluation. 

The  authors  oiake  no  claim  that  the  procedures  presented  herein  are 
necessarily  the  best  nor  the  only  ones  available— to  the  contrary  we  feel 
that  this  document  might  provide  merely  a  stepping  stone  tq>on  which  further 
iaprovements  may  be  made.  It  is  considered,  however,  that  with  the  factors 
of  cost,  time  and  personnel,  the  procedures  recosmended  will  provide  a  prac¬ 
tical  yardstick  by  which  management  may  make  an  intelligent  decision. 

LOGICAL  CONSIDERATIONS 

The  culmination  of  the  development  of  any  large  data  processing  system 
is  the  assembly  and  testing  of  the  prototype  system  against  the  specifications 
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for  which  it  is  designed.  Only  when  it  is  assembled  and  operated  as  a 
finished  product  can  the  compatibility  of  the  subsystems  with  one  another 
be  assessed  realistically.  Similarly,  it  is  only  when  assembled  and  operated 
that  the  compatibility  of  the  program  system  as  a  whole  with  the  computer 
equipment  can  be  assessed  realistically.  The  end  result  of  the  evaluation 
provides  the  groundwork  for  the  final  design  and  future  modification  of  the 
system.  If  the  results  show  that  the  operational  requirements  are  not  fully 
met,  then  the  requirements  must  be  altered  or  the  system  program  laid  out 
for  redesign  to  meet  the  requirements.  If  the  results  show  that  the  oper¬ 
ational  requirements  have  been  met,  then  the  results  must  be  documented  to 
provide  a  basis  for  future  design  change  and  expansion. 

While  there  exist  wide  differences  in  the  techniques  that  may  be  employ¬ 
ed  in  conducting  an  evaluation  of  this  type,  the  final  general  criteria  by 
which  the  worth  of  a  data  processing  system  is  measured  are  reasonably  con¬ 
sistent.  The  staff  or  management  personnel  who  make  use  of  the  information 
produced  are  interested  not  so  much  in  the  programming  steps  involved,  or 
the  evaluation  techniques  used,  but  are  more  concerned  with  a  determination 
that  the  outputs  of  the  system  are  as  follows: 

1.  Valid:  The  conputations  and  resultant  outputs  of  the  system  must  pro¬ 
vide  true  and  accurate  information  within  given  tolerances. 

2.  Current:  The  system  must  update  data  with  sufficient  frequency  to  insure 
that  all  computations  take  into  consideration  the  most  recent  factors  and 

reflect  them  in  the  outputs. 

3.  Complete:  The  outputs  should  provide  all  the  information  necessary  to 
permit  interpetation  and  analysis  by  the  user. 

4.  Accessible:  The  information  desired  of  the  system  must  be  quickly, 
easily,  and  directly  obtainable  to  the  user  without  recourse  to  complicated 
procedures  requiring  involved  or  lengthy  training. 

5.  Readable:  All  outputs  of  the  system  must  be  presented  in  a  form  that 
is  easily  understood  without  intermediate  translation  or  processing. 

6.  Usable:  All  output  information  should  be  pertinent  and  necessary  to 
the  requirements  of  the  user,  eliminating  the  waste  of  time  and  effort  in 
the  interpretation  of  information  that  is  of  no  interest  or  value. 
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The  criteria  given  above  are  necessarily  general,  since  this  procedure 
is  intended  to  be  applicable  to  most  real-time  systems.  It  must  be  stressed 
that,  in  each  particular  evaluation,  it  is  most  ijnq>ortant  for  management  to 
clearly  spell  out  its  system  output  requirements  in  detail  to  evaluating 
personnel.  An  evaluator  should  be  able  to  determine  whether  a  selected  input 
has  led  to  erroneous  output  in  all  cases  on  the  basis  of  the  criteria  for 
acceptance  and  rejection  vdilch  have  been  furnished  him  by  the  management 
involved;  i.e.,  vdiether  it  is  ’•right"  or  "wrong." 

The  logical  criteria  presented  here  are  essentially  designed  for  a  final 
checkout  of  a  system  and  program  using  the  outputs  of  the  system  as  a  basis 
for  this  determination.  Therefore  it  will  be  assumed  that  the  system  machin¬ 
ery  has  been  determined  to  be  reliable  by  operating  personnel  and  that  the 
operating  staff  is  sufficiently  well-versed  in  their  tasks  to  insure  that 
human  error  is  not  significant.  Previous  experience  has  indicated  that  these 
two  a8Suiif>tions  cannot  be  taken  lightly.  If  the  evaluator  finds  that  these 
sources  of  system  inefficiency  can  not  be  ruled  out  initally,  it  is  suggested 
that  he  consult  one  of  the  many  excellent  references  which  exist,  both  on 
machine  reliabUity  studies^  and  human  work  sam)ling.^ 

The  evaluator  should  insure  that  detailed  logs  are  kept  by  the  evaluating 
staff  so  that,  if  and  when  difficulties  occur  as  to  the  application  of  the 
logical  criteria  to  the  outputs,  they  may  be  readily  resolved  by  consultation 
with  the  appropriate  level  of  management.  Also,  if  specifications  or  inter¬ 
pretations  are  changed  during  the  course  of  the  evaluation— or  even  following 
x(».|[pp]7opriate  revisions  might  possibly  be  made  to  the  log  and  figures  ad¬ 
justed.  A  costly  repeat  trial  run  may  thereby  be  avoided. 

The  evaluator  also  must  know  or  be  given  an  appropriate  time  increment 
on  which  to  base  his  random  observations  during  the  real-time  sanple.  Some 
estimate  as  to  the  frequency  of  entrance  of  inputs  into  the  system  would  be 
of  much  help  on  this  question. 

Finally,  it  is  suggested  that  a  simulated  input  representing  each  type 
of  actual  input  be  entered  into  the  machine  prior  to  actual  real-time  testing 
for  two  reasons: 

1.  To  check  that  format  checks  in  the  program  are  working  properly  for  all 
types  of  input; 
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References  are  listed  on  page  27. 


4 


2.  To  assure  that  at  least  one  observation  is  present  on  each  type  of 
input.  As  will  be  discussed  later,  the  mean  frequency  of  occurrence  of 
some  type  (s)  of  input  may  be  so  low  as  to  rule  out  an  occurrence  of  an  input 
of  a  certain  type  during  the  actual  real-time  test.  In  this  case,  it  is 
advantageous  to  state  that  at  least  one  simulated  representative  input  of 
that  type  had  cleared  through  the  machine. 

STATISTICAL  CONSIDQIATIONS 

Though  the  evaluator  is  not  primarily  interested  in  measuring  the  effect 
of  time  upon  the  processing  of  inputs,  he  is  most  interested  in 
what  bias  it  may  contribute  to  the  reliability  estimate.  Randomisation  of 
input  selection  with  respect  to  time  insures  that  the  probability  of  select¬ 
ion  of  a  particular  activity  level  of  the  machine  and  particular  input  will 
be  the  same  as  the  probability  of  the  selection  of  any  other  activity  level 
and  input.  Any  systematic  effect  of  time  or  activity  level,  if  present,  is 
thereby  minimized  on  all  types  of  input  processing. 

Two  separate  random  saiqples  form  the  basis  for  the  statistical  inferences 
which  the  evaluator  must  make  in  support  of  his  conclusions.  The  first  sasple, 
or  "stage"  of  the  experiment,  is  made  from  a  large  collection  of  previous 
input  data  for  which  a  total  analysis  would  be  uneconomical  and  time-consuming. 
Since  the  real  time  data  processing  system  replaces  either  a  lesser  system 
or  hand  calculations  dealing  with  similar  inputs,  a  large  amount  of  previous 
input  data  can  and  must  be  made  available  to  the  evaluator  for  random  saspling 
for  representative  criteria. 

The  second,  or  "actual"  sanple  is  made  on  location  during  a  relatively 
short  continuous  real  time  run,  with  randomization  with  respect  to  real  time. 
For  the  purpose  of  establishing  validity,  it  is  mandatory  to  assure  that 
the  actual  real  time  sanple  which  one  takes  for  the  reliability  measurements 
is  representative  of  the  total  population  of  inputs  which  one  might  expect 
to  enter  the  system  over  an  extended  period  of  time.  The  criteria  which  are 
used  for  this  determination  of  representativeness  of  the  second  sasple  are 
the  frequency  estimates  of  the  various  types  of  input  over  a  long,  continuous 
period  of  operation. 
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A.  FIRST  STAGE;  ESTIMATION  OF  FREQUHiCY  CRITERIA 

As  stated  previously,  the  first  stage  of  the  evaluation  establishes, 
over  as  long  a  period  as  possible,  the  relative  frequency  of  each  input 
into  the  system.  Considering  each  input  classification  separately,  each 
randomly  selected  input  will  either  belong  to  this  input  classification 
or  not.  Thus,  it  is  possible  to  consider  the  estimates  of  each  input  mean 
frequency  as  distributed  binomially.  Average  values  will  be  ^proximately 
normally  distributed  with  increaiiing  sanple  size  even  when  the  observations 
are  from  a  skewed  distribution  of  individual  items.  Thus,  we  are  able  to 
use  the  normal  curve  statistics  to  advantage,  regardless  of  the  shjq)e  of 
the  distribution  from  which  the  samples  are  drawn.  With  the  establishment 
of  the  saiqple  x  (average  value  for  each  input),  if  we  can  derive  and  use 
some  standard  deviation  measure  for  a  distribution,  we  can  also  establish 
upper  and  lower  confidence  limits  for  this  average  value.  In  estimating 
this  average  value,  it  is  far  better  to  make  a  random  saiqsle  from  a  long, 
continuous  period  of  operation  than  to  use  all  inputs  from  a  block  portion 
of  a  long  period  for  consideration,  since  biases  or  peaks  toward  a  predom- 
inence  of  one  or  more  inputs  are  more  likely  to  remain  undetected  during 
shorter  period  of  operation.  If  cost  and  time  allow,  a  complete— -rather 
than  random— selection  of  inputs  over  an  extended  period  of  time  should 
be  even  more  satisfactory  for  estimation  of  the  frequencies  and  associated 
confidence  limits  at  this  stage,  the  evaluator  Quite  sure  that  the 
observations  USL 

For  clarity,  the  following  hypothetical  exanple  is  presented:  analysis 
of  a  real-time  data  processing  system  discloses  five  distinct  types  of  input, 
A,  B,  C,  D,  and  E.  One  thousand  inputs  of  these  types  are  selected  randomly 
from  a  5-week  period.  The  results  of  this  sanpling  ai^  shown  in  the  follow¬ 
ing  table : 
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Input 

Week 

Totals 

B 

2 

3 

B 

5 

A 

32 

44 

29 

33 

178 

B 

98 

85 

100 

H 

99 

465 

C 

46 

38 

46 

33 

223 

D 

25 

27 

23 

B 

33 

123 

E 

Jt, 

B 

OI. 

Total 

201 

196 

203 

199 

.  201 

1000 

The  8«|)le  relative  frequencies  are: 

«  "  0.178 

b  =  0.465 

c  =  0.223 

d  =  0.123 

e  =  0.011 

Though  these  are  the  best  estisuites  available,  they  are  at  best  s«|>le 
values  and  it  is  wise  to  gain  sone  idea  of  how  accurate  the  estisiation 
actually  is.  Either  exact  v  confidence  limits  or  the  approximation 
provided  by  Student’s  t-distribution**  could  be  used  for  this  determination. 
For  sasples  of  this  sise  and  larger,  the  ^proximation  based  on  the  t- 
distribution  appears  quite  satisfactory  and  is  far  easier  to  coB|>ute.  These 
limits,  which  can  be  found  in  most  work  saaf>ling  and  statistical  texts,  are 


^ee  Reference  3,  page  698. 
^«^See  Reference  4,  page  77. 
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where 

n  =  total  nunber  of  observations  in  the  sasiiple 
a  =  desired  critical  region  for  100  (1  -  a)  56  confidence  limits 
X  =  sanf>le  mean 

t  =  Student*s  t  with  (n  -  1)  degrees  of  freedom 

The  particular  confidence  level  (defined  by  a)  which  is  desired  will 
probably  differ  with  each  particular  experiment.  It  is  recomnended  that 
an  a  less  than  or  equal  to  0.05  be  used. 

For  95  percent  confidence  limits  on  the  true  mean  frequency  6^  of 
input  A  we  have  (a  =  0.05) 


=  0.178  +  (1.96)  (0.013002)  =  0.203 
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Similarly, 

0^^  =  0.178 

(1.96)  (0.013002)  =  0.154 

So  that,  in 

repeated  sanf>ling. 

pf  -  <0  <0) 

'a  a  A^ 

=  P(0.154  <  0.203)  =  0.95 

Similar  calculations  for  the  remaining  input  types  yield 

<D| 

00 

n 

o 

. 

^  =  0.434 

Qq  =  0.249 

^  =  0.197 

• 

o 

II 

®jj  =  0.103 

0£  =  0.017 

0£  =  0.005 

Since  the  maxiimim  value  for  x  (1  •>  x)  is  1/4  (when  x  =  1/2),  we  see 
that  the  larger  deviations  about  a  saople  frequency  will  occur  when  the 
frequency  approaches  1/2  and  that  smaller  deviations  will  occur  when  the 
frequency  is  near  0  or  1.  Also,  since  the  limits  are  inversely  pro¬ 
portional  to  /ii,  they  will  be  closer  together  (or  the  estimates  more  accurate) 
as  the  sssgile  size  n  is  increased.  For  this  reason,  the  largest  possible 
saB|>le  must  be  taken  in  the  first  stage. 

B.  SECOND  STAGE:  REPRESEMTATION  AND  RELIABILITY 

With  estimates  of  frequencies  and  associated  confidence  limits  deter¬ 
mined,  the  actual  real  time  saBf>le  must  be  devised  for  the  determination  of 
the  reliability  of  the  system.  The  organization  of  this  second  stage  of 
saiif>ling  consists  of  the  following  steps: 

1.  A  determination  of  the  sang)le  size  necessary. 

2.  A  randomization  of  input  selection  with  respect  to  real  time  (or 
activity  level  of  the  system). 
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3.  When  the  sample  is  accepted  as  representative  ,  a  determination  of 
the  reliability  of  the  total  system  and— if  possible— each  type  of  input, 
with  appropriate  confidence  limits. 

Considered  together,  the  frequencies  from  saiif>le  1  are  estimates  of 
the  actual  mean  frequencies  8^  of  the  multinomial  distribution  with  fre¬ 
quency  fUncticHi 


^  *  *  *  * 


V  *b’  V 


08 


This  formula  denotes  the  probability  that  input  A  will  occur  times, 
input  B  will  occur  x^  tisws 
observations  of  the  second  saBy>le,  i.  e.. 


,  ,  and  input  K  will  occur  x^^  tiaws  in  n 


Further,  it  is  assuamd  that 

1) .  the  theoretical  mean  frequencies  sum  to  one; 

2) .  the  input  types  are  mutually  exclusive;  and 

3) .  the  second  sas|>le  observations  are  stochastically  independent. 

This  latter  aasuBf>tion  usually  necessitates  a  random  selection  of  the 
iiq;>uts  to  assure  that  each  single  input  has  an  equal  change  of  selection 
for  the  second  sasple  estiawtes  of  input  type  frequency. 

Our  criterion  for  representativeness  is  the  Chi-square  *tCoodness  of 
Fit”  test.  It  is  well -suited  for  the  multinoadal  distribution,  as  it  affords 
a  single  test  for  the  representativeness  of  all  frequency  estiauites. 

The  formula  is 


'*When  the  second  stage  saBq>le  is  rejected  as  unrepresentative,  the  experinmnt 
reverts  to  the  first  stage.  A  more  recent  first  sasple  shoxild  be  taken,  of 
a  larger  size  if  possible.  Frequency  estiawtes  and  confidence  limits  must 
be  reformulated  for  the  iag)ut  types,  since  the  rejected  saaqile  furnishes  us 
with  significant  evidence  that  the  relative  frequencies  are  changing. 

A  close  coaparison  of  the  frequencies  obtained  in  saaple  2  end  the  confidence 
liaiits  established  in  saaple  1  should  give  a  good  indication  of  which  input 
frequencies  are  changing. 
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where 

k  =  number  of  input  typos 

3.°=  theoretical  frequency  of  occurrence  of  input  i  as  estimated 
in  sample  1 

x^-  number  of  occurrences  of  input  i  in  saiif>le  2 

f  =  degrees  of  freedom 

n  =  number  of  observations  in  sanple  2 

Examination  of  the  derivation  of  this  formula  reveals  that  the  sampling 
2 

distribution  of  X  tends  to  a  limiting  distribution  independent  of  the 
probability  function  (in  this  case  the  multinomial),  depending  merely  on  the 
parameters  (in  this  case  the  3^)  which  are  to  be  estimated  from  the  sanple. 

Since  this  test  eng>loys  the  normal  i4>proximation,  it  is  necessary  to 
have  a  second  sample  size  n  large  enough  for  such  an  approximation  to  be 
valid.  Fisher  recommends  that  n  be  large  enough  so  that  each  n3^  >  5 
Therefore,  the  minimum  sanple  size  is  obtained  by  taking  the  smallest  fre¬ 
quency  from  sanple  1,  3  and  letting 

n  =  5  / 

Noting  that  n  must  be  an  integer,  round  off  to  the  next  highest  integral 
value. 

Referring  to  the  hypothetical  problem  again,  determine  the  random 
sample  size  necessary  to  validate  the  normal  approximation  for  sanple  2. 

The  smallest  input  frequency  is 

Solving  for  n, 

n  =  5  /  0.011  =  454.545 


^^See  Reference  5,  page  417. 
'ws^See  Reference  4,  page  135. 
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Rounding  off  to  the  next  higher  integer  value,  n  =  455.  Thus,  455  randoia 
observations  are  necessary  in  saiig>le  2  to  assure  the  validity  of  the  Chi- 
squared  "Goodness  of  Fit"  test  for  representativeness.  On  the  basis  of 
the  niBiiber  of  observations  necessary  for  the  saii|)le,  the  randomization  on 
time  increments  can  be  derived  as  sho%m  in  Section  IV  on  evaluation  procedure. 

We  illustrate  the  use  of  the  Chi-squared  test  by  the  following  exaii|)le: 
Siq>pose  455  inputs  are  randomly  selected  in  time,  yielding  the  results 
illustrat  a  in  the  folj.wttl.  ^«hle. 


Input 

Number  (x^^) 

Frequency  (ssaple  2) 

A 

87 

0.191 

B 

220 

0.484 

C 

92 

0.202 

D 

53 

0.116 

E 

3. 

0.007 

Since  there  are  five  input  types,  we  compute  a  Chi-square  with  four 
degrees  of  freedom.  In  this  example 


x2  = 


(x^  -  (455)  e^)2  .  (455)  0^)2  .  (455)  e^)2 


(455)  e 


(455)  0 


B 


(455)  0, 


^  -  (455)  e^)= 


(455)  0, 


(455)  0. 


(87  -  (455)  (0.178)  Y 
(455)  (0.178) 


(220  -  (455)  (0.465)  Y 
(455)  (0.465) 


(92  -  (455)  (0.223)  )^  (53  -  (455)  (0.123)  )^  (3  -  (455)  (0.11) 

+ _ +  _ +  _ 


(455)  (0.223) 


(455)  (0.123) 


(455)  (0.011) 


=  2.6247 
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Since  ®  2.75  (P  {  <  2.75  3  =  0.40)  with  four  degrees  of  freedom, 

the  value  obtained  from  the  san|>le  is  in  the  lower  40^  of  the  Chi-squared 
distribution  for  four  degrees  of  freedom.  Since  we  may  expect  values  greater 
than  this  more  than  60^  of  the  time,  there  are  no  grounds  present  for  re¬ 
jecting  the  hypothesis  that  sample  two  is  from  the  same  population  estimated 

2 

by  saiple  one.  If  the  sasple  Chi-squared  value  were  greater  than  Xq  == 
9.49  for  four  degrees  of  freedom,  we  would  have  significant  evidence  for 
rejecting  the  saBf)le  as  ^representative . 

Having  established  the  representativeness  of  the  saiiq>le,  the  final  step 
of  the  evaluation  is  to  obtain  a  reliability  estimate.  The  reliability 
measiurements  are  made  by  following  the  selected  inputs  through  the  machine 
processing  and  by  analyzing  the  outputs  resulting  from  the  selected  random 
inputs  on  the  basis  of  the  logical  criteria.  Since  the  outputs  resulting 
from  a  given  input  are  either  correct  or  in  error,  we  have  a  binomial  dis¬ 
tribution.  The  probability  that  there  are  not  more  than  x  erroneous  outputs 
resulting  from  a  total  of  n  inputs  is  therefore  given  by  the  cumulative 

binomial  distribution.  This  is  known  to  be  equivalent  to  an  incomplete  beta 

2  ^ 

integral,  as  is  the  v  or  Fisher’s  F  distribution  ,  which  is  widely  tab¬ 
ulated.  Since  our  second  sasple  will  necessarily  be  smaller  than  the  first, 
exact  confidence  liaiits  are  used  rather  than  those  afforded  by  the  approx¬ 
imation  with  the  t  distribution.  The  limits  on  the  actual  frequency  of  error 

2 

of  the  population  of  inputs  from  which  we  sn^le  are  based  on  the  v  or 
Fisher’s  F  distribution  and  are 


(Xq  +  1)  (f^,  f2) 


f^  =  2(Xq  +  1) 

f2  =  2(n  -  Xp) 


■J^See  Reference  3,  pages  672-675. 
**See  Reference  3,  page  698. 
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and 


Xq  +  (n  -  Xjj  +  1)  (f/,  f/) 

=  2  (n  -  Xp  +  1) 

^2 


where 

n  =  total  observations  in  saiif>le  2 

Xq=  total  number  of  inputs  resulting  error  in  sample  2 

p  .  p  =  1  -  a,  which  defines  100(1  -  a)%  confidence  limits 

=  P  {  X  <  v^p  (fj^*,  fj 
2 

Pj^  =  P  {  X  <  V^p^  (fj^,  fj)^ 
where  x  is  a  random  variable  from  0  to  •• 

Assuiae  now  that  the  455  inputs  selected  in  the  exaiif>le  have  been 


analysed  with  respect  to  their  effect  on  the  system  as  pictured  by  the 
outputs  which  are  influenced  by  them.  The  results  may  be  sumnarised  in 
the  following  table: 


Input 

Number 

(X.) 

Number 

Correct 

Number 

in  Error  (x^) 

E 

R 

A 

87 

85 

2 

0.0230 

0.9770 

B 

220 

220 

0 

0.0000 

1.0000 

C 

92 

91 

1 

0.0109 

0.9891 

D 

53 

49 

4 

0.0755 

0.9245 

E 

3 

3 

0 

0.0000 

1.0000 

Overall 

455 

448 

7 

0.0154 

0.9846 

-::-E  =  frequency  of  error  (as  estimated  by  sample  2) 
-;KfR  =  (1  -  E)  reliability  (as  estimated  by  sample  2) 
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Ninety-five  percent  confidence  limits  on  the  overall  error  frequency 
are  as  follows:  (P^  =  0,975,  Pj^  =  0.025) 

*  1)  ’^0.975 
448  +  (7  +  1)  y^a.97S 
8(1.62) 

=  -  =  0.0315 

448  +  8(1.82) 

7  7 

=  - 5 -  =  -  =  0.0062 

7  +  (449)  v^  Q  g^5  (898,14)  7  +  (449)  (2.49) 

true  overall  reliability  is  between 
=  1  -  E  =  0.9938 


E 

Thus,  the 
R 


and  R  =  1  -  E  =  0.9685 

in  95^  of  all  samples  of  this  size.  It  is  well  in  general  to  obtain  a  high 
percentage  of  confidence,  but  the  reliability  confidence  limits  R  and  R 
will  be  closer  together  at  the  lower  confidence  percentages.  This  should 
be  remembered  if  the  confidence  interval  for  the  evaluation  is  required 
to  be  within  a  certain  numerical  tolerance,  regardless  of  the  confidence 
percentage . 

Limits  may  be  computed  for  each  input  type  if  desired.  For  this  exaaple, 
one  has  the  various  frequencies  of  error  with  95^  confidence  limits  as  follows: 


E.  -  0.0852 
A 

Eg  ■  0.0166 

E  -  0.0591 

Eg  “  0.1820 

-  0.7076 
is 


E^  -  0.0028 

Eg  “  0.0000 

^  =  0.0003 
Eg  “  0.0209 

Eg  =  0.0000 
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Several  items  may  be  noted  concerning  the  preceding  lijnits.  Note  that, 
besides  offering  a  more  exact  derivation  of  confidence  limits  for  the  bi¬ 
nomial  distribution,  these  limits  also  offer  a  convenient  expression  for 
confidence  limits  on  frequency  of  error  E  even  when  np  errors  actually  occur 
during  the  actual  evaluation.  £  is  iinnediately  seen  to  equal  zero  when 

-  0,  but  examination  discloses  that  E  is  never  zero.  If  the  normal  approx¬ 
imation  or  t-test  approximation  were  used  in  this  case  no  limits  could  be 
drawn  at  all. 

Also,  the  role  of  the  number  of  observations  of  each  input  is  clearly 
seen  in  these  limits,  as  only  the  limits  for  inputs  appear  to  be  of  any 
use.  Inputs  D  and  E  have  so  few  occurrences  as  to  make  the  limits  too  far 
4>art  to  be  useful  at  the  95^  level.  What  is  ijiportant  in  the  foregoing 
calculations  is  that  the  reliability  may  be  estimated  within  fairly 

close  bounds  even  if  some  of  the  types  of  inputs  appear  quite  infrequently 
during  the  second  sanple.  Although  it  is  advisable  to  gain  an  actual  case 
of  the  occurrence  of  each  type  input  during  the  actual  real  time  sampling, 
the  absence  of  one  or  more  types  does  not  at  all  negate  the  drawing  of  con¬ 
fidence  limits  for  the  reliability  estimate. 

It  is  conceivable  that  in  many  instances  no  errors  at  all  will  occur 
during  the  real  time  evaluation  (swf>le  two).  It  is  well  to  note  that  con¬ 
fidence  limits  can  be  drawn  for  this  very  desirable  phenomenon  also.  In  our 
example,  with  no  errors  in  455  inputs,  the  iq>per  95^  confidence  limit  on  the 
frequency  of  error  is 

455  +  v^  Q  (2,910) 

3.70 

=  -  =  0.0081 

458.70 

The  lower  limit  E  is,  of  course,  zero.  This  means  that  the  evaluator  could 
state  with  a  95;^  confidence  that  the  overall  reliability  of  the  system  would 
be  between 

R  =  1  -  E  =  1.0000  and  R  =  1  -  E  =  0.9919 

in  repeated  sanpling  on  the  basis  of  this  sanple. 
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THE  CONDUCT  OF  THE  EVALUATION 


The  following  procedural  steps  present  the  chronological  order  by  which 
the  evaluation  should  take  place  and  will  serve  also  as  a  suggested  outline 
for  the  final  report.  In  a\aiinary>  both  the  evaluation  and  the  final  report 
of  that  evaluation  c<msist  of  the  following  basic  steps: 

A.  Preliminary  orientation  with  the  system  to  be  evaluated. 

STAGE  ONE 

B.  Assembly  of  a  large  continuous  segnent  of  previous  inputs. 

C.  Estimation  of  input  frequencies  with  associated  confidence 
limits. 

STAGE  TWO 

D.  Random  sasgiling  of  actual  inputs  during  actual  operation  on 
the  basis  of  real  time. 

E.  Establishing  the  representativeness  of  the  real  time  random 
ssiif)le. 

F.  Estimation  of  system  reliability  with  associated  confidence 
limits. 

The  purpose  of  this  section  is  to  permit  the  conduct  of  the  evaluation 
with  relatively  inexperienced  personnel;  accordingly,  an  attempt  has  been 
made  to  reduce  the  material  ccmtained  herein  to  its  sijiplest  state.  Un¬ 
fortunately,  however,  the  results  of  the  evaluation  are  based  on  certain 
statistical  theory,  and  it  is  stressed  that  each  and  every  step  that  is 
indicated  be  strictly  adhered  to.  The  formulas  that  have  been  used  should 
require  merely  substitution  therein  and  do  not  necessarily  presi^pose  ex¬ 
tensive  mathematical  knowledge.  If  the  formulas  are  not  clear,  however, 
they  should  be  clarified. 

A  breakdown  of  the  foregoing  basic  steps  results  in  the  following 
evaluation  procedure: 

A.  PRELIMINARIES 

Step  1 .  Broad  Statement  of  System: 

Reduce  to  written  form  a  broad  sunmary  of  the  system  to  be  eval¬ 
uated.  This  may  be  done  by  a  condensation  of  the  specifications  combined 
with  a  brief  description  of  how  the  system  is  designed  to  meet  these  spec¬ 
ifications. 
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step  2.  Detailed  Statement  of  System  with  Regard  to  Tvpea  of 
Inputs  and  Outputs: 

Though  the  evaluator  should  have  a  general  picture  of  the  flow 
of  information  through  the  system,  the  statistical  nature  of  the  evaluation 
makes  it  imperative  that  he  be  especially  familiar  with  the  types,  formats, 
and  tolerances  involved  in  the  inputs  and  outputs;  and  that  he  demonstrate 
this  familiarity  in  the  final  report. 

Step  3.  A  Meeting  with  Operating  Perronnel  to  Determine  in 
Detail  what  Satisfies  Logical  Criteria  for  Output ; 

Here  the  general  criteria  presented  in  the  logical  section  should 
become  concrete  to  the  evaluator.  He  should  be  able  to  tell  whether  or 
not  any  selected  output  satisfies  the  six  general  logical  categories: 
Validity,  Currentness,  Coapleteness,  Accessibility,  Readability,  and 
Usability.  From  discussions  with  operating  personnel,  he  should  be  able 
to  compare  original  specifications  against  current  operational  requirements 
and  discuss  any  discrepancies  in  the  final  report. 

Step  4.  Establishment  of  Input  Categories: 

All  of  the  inputs  to  the  system  must  be  firmly  typed  and  cat-> 
egorized.  In  most  cases  this  will  have  been  accomplished  by  the  programm¬ 
ing  group  in  their  assignment  of  an  individual  identification  code  to  each 
input.  If  this  has  not  been  done,  then  the  evaluator  should  assign  his  own 
identification  to  remove  the  possibility  of  ambigu-Lty. 

STAGE  ONE 

B.  ASSEMBLY  OF  A  LARGE  CONTINUOUS  SEGMENT  OF  PREVIOUS  INPUTS 

Step  5.  Assemble  Largest  Continuous  Segment  of  Previous  Inputs 

The  main  stress  here  is  on  the  word  "continuous."  In  the  first 
stage  sanple,  time  is  only  inportant  insofar  as  there  are  no  breaks  in  time 
present  during  the  period  when  the  inputs  are  conpiled.  The  total  inputs 
may  be  arranged  in  any  arbitrary  order  for  random  sampling,  keeping  in 
mind  that  the  greater  the  number  of  observations,  the  more  accurate  the 
frequency  estimates  for  the  various  types  of  input. 
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step  6.  Random  Sanallng  -  Stage  One:  ^ 

The  uae  of  the  random  8«i|)ling  ntanbera  in  this  aanple  ia  outlined 
aa  follows: 

a.  Aaaenble  iiq|>uts  in  any  order. 

b.  Beginning  at  the  iq;>per  left-hand  comer  of  a  two-figure  random 
ntimber  table,  count  this  many  Inputs  and  aelect  the  last  one  counted. 

Remove  this  input  and  note  its  type. 

c.  Select  the  next  number  below  and  proceed  aa  above,  column  by 

column. 

d.  When  the  end  of  the  inputs  is  reached,  begin  at  start  again; 
continue  as  long  as  possible. 

e.  Total  tallies  for  each  input  type  and  for  entire  8aB|>le. 

C.  ESTIMATION  OF  INPUT  FREQUENCIES  WITH  ASSOCIATED  CONFIDENCE  LIMITS 
Step  7.  Uae  of  Fonmilaa  for  Staae  One; 

a.  Ihe  sample  relative  fre({uency  for  each  type  input  is  obtained 
by  taking  the  ratio  of  the  total  number  of  inputs  of  a  certain  type  to  the 
total  number  of  all  Inputs  in  the  random  sample.  For  example,  in  the  hypo¬ 
thetical  problem  presented  in  the  statistical  section  of  this  report,  input 
A  occurred  178  tiams  in  1000  inputs.  Thus,  the  aanple  relative  frequency 
for  ijq;>ut  A  is 

178 

•  “  1000 

b.  Confidence  limits 


0  =  X  + 


0  =  X  - 


0.178 


/  X  (1  - 

V  — 

/  X  (1  -  ■ 

V— 


*See  Reference  6,  pages  92-97. 
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In  the  formulas  for  6,  the  upper  cmifldence  limit,  and  3,  the  lower  confidence 
limit,  each  input  must  be  considered  separately  and  x  represents  whatever 
particular  sample  relative  frequency  with  which  we  are  concerned;  e.g.,  x 
is  a  when  we  are  discussing  input  A.  The  t^  value  is  obtained  from  Hald*s 
Table  IV  by  reading  (n  -  1),  one  less  than  the  total  number  of  inputs  in 

the  sample,  on  the  vertical  scale;  and  a  =  2  (1  -  P),  where  P  is  the  desired 

percentage  of  probability  on  the  bottom  scale. 

For  90^  confidence  limits,  use  a  «  10  {%)  (1.645) 

95^  confidence  limits,  use  a  =  5  (^)  (1.960) 

98^  confidence  limits,  use  a  *=  2  {%)  (2.326) 

99^  confidence  limits,  use  a  «  1  (%)  (2.576) 

99.8^  confidence  limits,  use  a  «  0.2(^)  (3.090) 

99.9^  confidence  limits,  use  a  »  O.l(^)  (3.291) 

Values  given  in  parentheses  may  be  used  accurately  for  t-values  for  samples 
of  1000  or  more.  Linear  interpolation  will  suffice  for  intermediate  values 
on  the  vertical  scale. 

c.  It  is  statistically  iBf>ortant  for  the  desired  percentage  of 
confidence  to  be  selected  prior  to  the  determination  of  the  sa^>le  relative 
frequencies . 

d.  Convenient  tables  are  available  for  the  determination  of  the, 
square  root  term  in  the  formulas  for  6  and  6. 

With  the  input  frequencies  and  associated  confidence  limits  obtained, 
the  first  stage  sasq)ling  comes  to  a  close.  It  now  becomes  the  responsibility 
of  the  evaluator  to  rule  out  certain  causes  of  error  in  the  actual  machine 
run  so  that  the  Stage  Two  sample  will  properly  reflect  the  reliability  of 
the  program  versus  specifications  and/or  operational  requirements  on  the 
basis  of  the  logical  criteria.  This  elimination  of  undesired  sources  of 
error  consists  of  three  steps. 


^See  Reference  6,  page  39. 
•JH^See  Reference  6,  pages  84-87 
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step  8.  Deteminetlon  of  Reliabmty  of  Machinery  end  Operation! 

The  machinery  must  exhibit  a  previous  percentage  of  downtime  small 

enough  for  the  evaluator  to  be  reasonably  sure  that  a  breakdown  will  not 

occur  during  the  period  required  for  the  actual  real  time  sany>le.  Operating 

personnel  should  be  so  well-^rsed  in  their  duties  that  no  appreciable  errors 

in  input  preparation  or  output  representation  will  occur  during  the  Second 

Stage  saB|>ling.  If  it  iq)pears  that  these  factors  must  be  tested,  procedures 

1  2 

for  such  evaluation  exist  *  and  should  be  consulted. 

Step  9.  Test  Each  Type  Input.  Format  Checks,  and  Tolerances; 

a.  Run  one  exan|>le  of  each  type  input  through  the  System  to  de- 
tennine  that  at  least  one  exaiig>le  of  each  type  input  is  acceptable  to  the 
system  and  results  in  a  correct  output. 

b.  All  checks  for  proper  format  and  tolerances  must  be  checked 
for  possible  prograaning  errors  before  beginning  the  actual  evaluation  run. 

Step  10.  Time  Allowance  for  Getting  Svatam  Undervav  i 

Follo«dng  machinery  downtime  or  operational  checkout,  real  time 
systems  often  require  a  period  of  time  to  reach  normal  capacity  and  oper¬ 
ation.  Since  the  distribution  of  iiqnit  data  during  this  period  may  be 
highly  unrepresentative  of  normal  operating  conditions,  it  is  well  to  post¬ 
pone  the  initiation  of  random  input  selection  until  this  period  has  been 
passed  sucess fully  and  the  system  is  operating  "normally”.  The  evaluator 
should  detenaine  if  such  a  condition  exists  for  the  particular  system  with 
which  he  is  involved  by  personal  familiarity  or  consultaticm  with  operating 
personnel.  Since  such  a  condition  will  result  whenever  a  machinery  shut¬ 
down  or  major  operational  error  occurs,  the  actual  real  time  sampling  should 
be  started  over  again  after  such  an  occurrence. 


STAGE  TWO 

D.  RANDOM  SAMPLING  OF  ACTUAL  INPUTS  DURING  ACTUAL  OPERATION  ON  THE  BASIS 
OF  REAL  TIME 

Step  11.  Determine  th*  Mini-m.  Mecessarv  Sixe  for  the  Second  Sample ; 

a.  Select  the  input  with  the  smallest  frequency  as  determined  in 

Step  7a. 

b.  Divide  this  frequency  into  5, 
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c.  Round  the  resultant  quotient  off  to  the  next  hipest  integer; 
this  Integer  is  n,  the  minimum  saople  size  for  sanple  two.  (If  n  <  100, 
set  n  100.) 

With  the  minimum  sample  size  thus  determined,  we  can  now  proceed  to 
distribute  these  sample  observations  over  a  period  of  actual  running  tiam 
in  a  random  fashion. 

Step  12.  Random  Sailing  on  the  Basis  of  Tiaie  -  Stage  Two; 

a.  Deciding  on  a  Time  Increment.  For  this  procedure,  it  is  most 
desirable  to  have  sooie  estisiate  as  to  how  often  inputs  tdll  enter  the 
system,  so  that  randomization  can  be  based  on  the  average  length  of  system 
"waiting”  tine  between  inputs.  This  estimate  may  be  obtained  from  the  times 
of  previous  inputs  used  in  Stage  one  saa|>ling  or  may  be  prq>osed  by  operating 
personnel.  By  the  use  of  the  random  number  table,  we  are  assured  that  the 
observations  used  in  this  second  saB|)le  will  be  random;  picking  a  realistic 
increment  aids  in  attaining  the  shortest  possible  sampling  period  necessary 
to  obtain  the  number  of  actual  observations  required.  Periodic  display  or 
other  output  characteristics  may  necessitate  the  selection  of  a  more  in¬ 
efficient  increamnt  on  which  to  base  the  random  observations.  The  impor¬ 
tant  points  here  are  that  an  increawnt  be  selected  and  that  it  be  as  close 

as  possible  to  the  average  length  of  system  waiting  time. 

b.  Randomization.  Siq>pose  the  increment  decided  on  was  6  minutes. 
By  placing  a  decimal  point  before  each  number  of  the  random  nuaber  table, 

we  have  iamediately  7500  random  fractions  between  0  and  1,  with  mean  approx¬ 
imately  0.50.  Multiplying  each  of  these  numbers  by  2  times  6,  or  12,  as  they 


^f  course  it  is  possible  here  to  determine  a  minimum  sample  size  that  is 
actually  larger  than  the  original  extended  sasple.  A  very  samll  input 
frequency  for  a  certain  type  will  do  this.  In  this  case  two  alternatives 
are  open  to  the  evaluator: 

1.  He  may  choose  to  drop  the  very  infrequent  input  from  consideration 
of  the  minimum  second  sample  size.  In  this  case,  he  must  also  drop  it  from 
use  in  the  Chi-square  "Goodness  of  Fit”  test  also.  This  alternative  should 
be  used  when  the  next  most  infrequent  input  leads  to  a  considerably  more 
economical  sanple  size. 

2.  He  may  choose  to  pool  several  very  small  frequencies  into  one  input 
class  and  use  the  total  groip  frequency  (if  it  is  still  the  smallest)  for 
the  determination  of  n.  In  this  case,  it  is  most  important  that  this  group 
identification  be  maintained  during  the  subsequent  "Goodness  of  Fit"  test. 
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are  selected  vill  merely  change  the  table  to  7500  random  numbers  between 
0  and  12,  and  setting  the  mean  approximately  at  6,  which  was  the  prede¬ 
termined  waiting  time.  We  follow  the  steps  presented  in  Stage  One  for 
selection  of  the  random  numbers  and  multiply  each  number  by  twice  the 
average  waiting  time  to  determine  the  times  at  which  observations  will 
be  made.  Using  the  same  table  as  in  Stage  One,  we  assume  the  actual 
sample  begins  at  1200.  We  round  off  to  the  ner"*  ‘est  minute. 

First  number  selected:  IS 
12  X  0.15  =  1.80.  Round  off  to  2 
Select  first  input  entered  into  system  after  1202. 

Second  number  selected:  85 
12  X  0.85  -  10.20.  Round  off  to  10 

Select  next  input  entered  into  system  10  minutes  after  first. 

Third  number  selected:  47 
12  X  0.47  =  5.64.  Round  off  to  6 

Select  next  input  entered  into  system  6  minutes  after  second. 

Etc. 

Proceed  from  latest  time  calculated.  Continue  until  minimum  number  of  ob¬ 
servations  is  compiled.  This  nund)er  of  observations  is  the  "n”  obtained 
in  Step  11. 

E.  ESTABLISHING  THE  REPRESENTATIVENESS  OF  THE  REAL  TIME  RANDOM  SAMPLE 
Step  13.  Determination  of  Representativeness: 

The  following  procedure  concerns  the  use  of  the  "Goodness  of  Fit” 

■J'r 

formula 

k 

i  -  1 


■'f^See  Reference  b,  pages  40-43. 
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a.  Use  the  value  obtained  in  Step  11  for  n. 

b.  Use  estimates  obtained  in  Step  7a  for  to  3)(. 

c.  Count  numbers  of  Inputs  of  each  type  in  sample  two.  These 
numbers  are  Xj^  to  X|^. 

d.  k  is  the  number  of  types  of  input.  (If  the  modifications 
mentioned  in  footnote  of  Step  11 >  Stage  Two,  are  used,  k  will  be  reduced 
accordingly.)  Sum  as  indicated. 

e.  Select  a  confidence  interval.  Usual  percentages  are  greater 

than  0.95. 

f.  Find  f  on  vertical  scale. 

g.  Check  Chi-square  value  for  desired  percentage  and  given  f. 

Tf  it  is  larger  than  the  number  conq>uted  by  the  formula,  accept  the  saiq>le 
as  representative;  if  it  is  smaller,  reject.  If  sanple  two  is  rejected  as 
unrepresentative,  the  evaluation  reverts  to  Step  5  as  a  new  large  sanple 
must  be  taken  preferably  with  more  recent  data.  ^ 

h.  Prior  to  reverting  to  Step  5,  it  is  recomnended  the  — —  values 
be  conpared  with  their  respective  6  and  3  values  determined  in  the 

first  stage,  for  by  so  doing  it  might  be  possible  to  determine  which  input  (s) 
caused  the  discrepancy  which  in  turn  might  indicate  the  existence  of  a  sys¬ 
tem  change,  etc. 

F.  ESTIMATION  OF  SYSTEM  RELIABILITY  WITH  ASSOCIATED  CONFIDEMCE  LIMITS 
Step  14.  Use  of  Reliahilitv  Fonmlaa» 

The  following  steps  concern  the  use  of  the  formulas 


(Xq  +  1)  V^  (fj^,  f^) 

fl  =  2  (x^j  +  1) 

^2  "  ^  ■  *0^ 


# 
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m 

f 

\ 

and 


Xp  +  (n  -  Xq  +  1)  V  p  ) 

2 


=  2  (n  -  Xp  +  1) 

^2  “  ^ 

a.  Use  the  number  n  coiq>uted  in  Step  11  for  overall  reliability 
determination. 

b.  Count  total  number  of  inputs  resulting  in  error.  (This  is  not 
necessarily  the  saise  as  the  total  number  of  outputs  in  error,  since  one  in¬ 
put  may  lead  to  several  erroneous  outputs.  Nor  is  it  the  number  of  inputs 
which  affect  a  given  erroneous  output,  since  only  one  or  a  portion  of  the 
inputs  leading  to  that  output  may  have  led  to  the  error).  This  is  x^  for 
the  overall  reliability  estimate. 

c.  Count  the  nianber  of  inputs  tich  These  are  the  n*s 

for  reliability  estimate  for  each  type  input. 

d.  Count  number  of  inputs  of,,{jsli,^X2£  resulting  in  erroneous 
output.  These  are  the  x^^s  for  reliability  estimates  for  each  type  input. 

e.  Usual  values  for  and  are  as  follows: 

9056  confidence:  P^  =  0.05  P2  “  0.95 

9556  confidence:  P^^  =  0.025  P^  =  0.975 

9856  confidence:  P^^  =  0.01  P^  =  0.99 

9956  confidence:  P^^  0.005  ®  0»995 

2  * 

f.  Values  of  V  are  obtained  from  Hald's  Tables.  The  values  of 

2 

(1  ”  or  P^  respectively  will  determine  which  v  table  to  use;  f^^  is 
entered  horizontally  and  vertically.  Be  sure  that  f^  and  £2  are  not 
interchanged  since  v^  (f^,  f2)  does  not  usually  equal  v^  (f2,  f^). 

^ee  Reference  6,  pages  47-59. 
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g.  E  and  E  are  confidence  lijnits  on  the  frequency  of  error  on  the 
basis  of  the  second  satnple.  To  obtain  the  confidence  limits  for  reliability, 

R  =  1  -  E  =  upper  confidence  limit 

R  =  1  -  E  =  lower  confidence  limit 

h.  As  shown  in  the  hypothetical  exaiig>le  in  the  mathematics  section, 
reliability  estimates  for  a  particular  type  input  are  usually  not  very  ac¬ 
curate  unless  over  75  inputs  of  a  certain  type  have  been  selected  in  the 
random  sanple. 


2b 
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